The present disclosure generally relates to scene reconstruction, and more specifically to the virtual reconstruction of an articulated object within a scene.
Reconstructing a 3D scene from image sequences has been an important research question for several decades. In particular, structure from motion (SfM) techniques have been successfully used in a wide area of different applications such as localization, navigation, and image based modeling, up to reconstructing entire cities from unstructured image collections. However, these methods require the scene to be static.
Reconstructing deforming objects is a largely unsolved problem, but has a wide range of practical applications. Many deforming object, such as humans, animals and most human-made machines, move in an articulated way, or in other words, can be approximated by a set of piecewise rigid parts, connected by joints. Because of this articulated movement of humans, animals and most human-made machines there is significant interest in the task of articulated structure from motion (A-SfM).
One general solution to A-SfM is to use shape templates and deform them to match the observed images. This technique can be used when a shape template of the non-rigid object is available a priori, then a full 3D re-construction can be computed by deforming the template using motion priors to match the current observation. However, this method use detailed template and motion models and is not easily generalized.
A more generic approach is to track 2D points over a video, segment them into articulated parts and apply factorization methods. In order to segment the input trajectories, these methods generally minimize some energy function, usually exploiting point connectivity in 2D and modeling the object shapes in 2.5D or 3D.
Each of these, and other methods, has shortcomings which limit the usability of these techniques. Therefore, it is desirable to provide new systems and methods for automatically addressing such problems associated the reconstruction of an articulated object.